Limitation of ROC in Evaluation of Classifiers for Imbalanced Data

PurposeROC is a common evaluation metric for risk scores and classifiers mortality adverse events. However, ROC can provide misleadingly optimistic view of the performance classifier when data are imbalanced, example proportion events very small. This study illustrates ambiguity through case post-LVAD Right Heart Failure (RHF), utility Precision Recall Curve (PRC) as supplemental evolution tool.MethodsThis included 11,967 patients recorded in INTERMACS who received continuous-flow LVAD between 2006 2016 (mean age 57; 21% female 79% male) which incidence RHF was only 9% at 1 year (1,079 patients). These were randomly split into training set (60%) test (40%). A logistic regression developed using to predict RHF.ResultsROC Fig.1.A indicates good with Area Under (AUC) 0.83. contrast PRC Fig.1. B AUC 0.33 shows precision drops rapidly from (100%) 0.4 (40%) recall (sensitivity) increases slightly greater than 0%. The gray dot optimized point equalized sensitivity specificity approximately 76-77%. In contrast, corresponding same (76%) 23%. (See Fig.1.B) means that 23% predicted by this correct (True RHF). Thus, preponderance (77%) experience incorrectly classified (False enormous False not captured because calculation overwhelmed huge number observed denominator free RHF.ConclusionThe portray an overly-optimistic or score applied imbalanced data. informative insight about focusing on minority class. tool. RHF.